How ANFIS works

ANFIS, the Adaptive Neuro-Fuzzy Inference System, was proposed by Jang in 1993^[1]. The idea is to write a fuzzy rule base as a network, so that it can be trained with the tools of neural networks while staying a rule base you can read.

A rule

Take two inputs, $x_{1}$ and $x_{2}$ . A first-order TSK rule says:

if $x_{1}$ is $A_{i}$ and $x_{2}$ is $B_{i}$ then $f_{i} = p_{i} x_{1} + q_{i} x_{2} + r_{i}$

$A_{i}$ and $B_{i}$ are fuzzy sets. The part before then is the premise and decides where in the input space the rule applies. The part after it is the consequent: a small linear model that is valid in that region.

Four layers

flowchart LR
    X[inputs] --> M[1 membership]
    M --> R[2 rules]
    R --> N[3 normalization]
    N --> C[4 consequent]
    X --> C
    C --> Y[output]

Layer 1, membership. Each input is compared with each of its fuzzy sets. With a Gaussian set the result is

μ_{A_{i}} (x) = \exp (- \frac{(x - c_{i})^{2}}{2 σ_{i}^{2}})

a number between 0 and 1 that says how much $x$ belongs to $A_{i}$ . The centre $c_{i}$ and the width $σ_{i}$ are the premise parameters.

Layer 2, rules. The firing strength of a rule is the product of the memberships in its premise:

w_{i} = μ_{A_{i}} (x_{1}) μ_{B_{i}} (x_{2})

Layer 3, normalization. Strengths are divided by their sum, so they become weights that add up to 1:

{\bar{w}}_{i} = \frac{w_{i}}{\sum_{k} w_{k}}

Layer 4, consequent. Each rule proposes its linear answer, and the output is the weighted sum:

y = \sum_{i} {\bar{w}}_{i} (p_{i} x_{1} + q_{i} x_{2} + r_{i})

The coefficients $p_{i}$ , $q_{i}$ and $r_{i}$ are the consequent parameters.

Hybrid learning

The two kinds of parameter behave differently, and that is what makes ANFIS train well.

Look at the last formula with the premise parameters held fixed: the weights ${\bar{w}}_{i}$ are then just numbers, and the output is linear in every $p_{i}$ , $q_{i}$ and $r_{i}$ . The best consequents for the current premises can be found in one step with least squares, with no iteration.

The premise parameters sit inside the exponentials, so they are nonlinear and need gradient descent.

Each training epoch therefore has two passes:

Pass	Premise parameters	Consequent parameters
forward	fixed	solved by least squares
backward	gradient descent	fixed

Tip

Because half of the parameters are solved exactly at every epoch, hybrid training usually needs far fewer epochs than plain gradient descent on everything.

ANFIS Toolbox has this rule as its hybrid trainers, and also plain SGD, Adam, RMSProp and particle swarm optimization for when you want to compare. See A first model to try it.

Where it stops working

Layer 2 multiplies one membership per input. With two or ten inputs that is fine. With hundreds, a product of many numbers smaller than 1 goes to zero, and the normalization of layer 3 breaks down. That problem, and what to do about it, is the subject of Why fuzzy systems choke on many features.

J.-S. R. Jang, "ANFIS: adaptive-network-based fuzzy inference system," IEEE Transactions on Systems, Man, and Cybernetics, vol. 23, no. 3, pp. 665-685, 1993. doi:10.1109/21.256541 ↩︎