Why fuzzy systems choke on many features

A classic TSK fuzzy system works well with ten inputs and stops learning with a thousand. The reason is not the amount of data or the optimizer. It is one multiplication. This page shows the failure, lets you reproduce it, and explains the fix that highFIS is named after.

The multiplication

With Gaussian fuzzy sets, the membership of feature $d$ in rule $r$ is

μ_{r, d} (x_{d}) = \exp (- \frac{(x_{d} - m_{r, d})^{2}}{2 σ_{r, d}^{2}})

and the firing strength of the rule is the product over all $D$ features:

w_{r} = \prod_{d = 1}^{D} μ_{r, d} (x_{d}) = \exp (- \sum_{d = 1}^{D} \frac{(x_{d} - m_{r, d})^{2}}{2 σ_{r, d}^{2}})

Every factor is at most 1, so the product can only shrink as $D$ grows. That alone would be survivable. The trouble comes in the next step, when the strengths are normalized into weights:

{\bar{w}}_{r} = \frac{w_{r}}{\sum_{i = 1}^{R} w_{i}}

Written with the exponentials above, this is a softmax of the sums. Each sum has $D$ terms, so both the sums and the gaps between them grow with $D$ . A softmax fed with large gaps saturates: the rule with the smallest distance takes a weight of nearly 1 and every other rule gets nearly 0.

The system then behaves as if only one rule existed for each sample. The other rules receive no gradient, so they never improve. Training stalls.

See it happen

The demo places five rules at random and measures, over 200 random samples, how much of the weight the strongest rule takes. With five rules, a balanced system would give about 0.2 and a saturated one gives 1.

import numpy as np

rng = np.random.default_rng(0)
n_rules, n_samples = 5, 200


def softmax(z):
    z = z - z.max(axis=1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=1, keepdims=True)


print(f"{'D':>6} {'TSK':>8} {'HTSK':>8}")
for dims in (2, 10, 100, 1000):
    x = rng.normal(size=(n_samples, dims))
    centers = rng.normal(size=(n_rules, dims))
    # log of each Gaussian membership, with sigma = 1
    log_mu = -((x[:, None, :] - centers[None, :, :]) ** 2) / 2
    tsk = softmax(log_mu.sum(axis=2))    # product of memberships
    htsk = softmax(log_mu.mean(axis=2))  # geometric mean
    # how much of the output the strongest rule takes, on average
    print(f"{dims:>6} {tsk.max(axis=1).mean():>8.3f} {htsk.max(axis=1).mean():>8.3f}")

Press [run]. The TSK column climbs from about 0.43 with two features to about 0.99 with a thousand: one rule has taken over. The HTSK column stays near 0.2 the whole way.

The fix

The two columns differ in a single word of the code: sum became mean.

Instead of the product of the memberships, HTSK^[1] uses their geometric mean:

w_{r} = {(\prod_{d = 1}^{D} μ_{r, d} (x_{d}))}^{1 / D} = \exp (\frac{1}{D} \sum_{d = 1}^{D} \log μ_{r, d} (x_{d}))

Dividing by $D$ keeps what goes into the softmax at the same scale whatever the number of features. The order of the rules is unchanged, since the rule that was closest is still closest, but the weights no longer collapse onto it, and every rule keeps receiving gradient.

Note

An older workaround is to make the Gaussians wider as $D$ grows. It has the same effect on the softmax, but it ties the shape of the fuzzy sets to the number of features, and they stop meaning anything. Averaging in log space fixes the arithmetic and leaves the fuzzy sets alone.

In highFIS

This model is HTSKClassifier and HTSKRegressor. The library does the whole computation in log space, which also avoids the underflow you would get from multiplying a thousand small numbers.

HTSK is the starting point, not the only answer. The other model families attack the same problem with different aggregation rules, or by learning which features and rules to keep.

Y. Cui, D. Wu and Y. Xu, "Curse of Dimensionality for TSK Fuzzy Neural Networks: Explanation and Solutions," 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1-8. doi:10.1109/IJCNN52387.2021.9534265 ↩︎