Boolean functions for which the total number of units grows only polynomially in the number of bits of the input.
It is hard to find useful Boolean function that is not NERF.
K.Siu, V. Roychowdhury and T. Kailath, IEEE. Trans. on Computers, Dec., 1991:
Any symmetric Boolean function (in n varaibles) can be computed with O(r(n)) threshold gates in a depth-3 network. Any Boolean function (in n varaibles) can be computed with O(2n/2) threshold gates in a depth-3 network.
Capacity paradox: Consider N input bit and 1 output bit. There are possible input patterns and therefor possible rules.. For example when N = 30 the possible rules are as many as 2109!
However reasonable rules are specified by no more than Nk bits for some small k. We should only consider NERFs, not general Boolean functions.
Generalization
Exploring the relationship between some response y and a number of predictor varaibels x=(x1,x2,...,xk). Namely, find a function g such that
is as small as possible.
If the structure of g(x) is given then the problem reduces to the determination of the parameters.
Example: g(x) is a polynomial?
Linear Regression Model:
Response Surface Model:
Neural Network Model