Hiding explanation data does not stop theft of tabular models

Researchers studied model extraction, which is a copy of a model made with queries and the answers. They looked at tabular classifiers (models that sort rows of data) behind explanation APIs (services that say why a model made a decision). Their attack, Cutpurse, gives a copy that is 16.3 percentage points nearer to the target. The service withholds Lime bin edges.

How it works

The authors divide an explanation into two channels. One ranks the features that are important. One gives values to test. Lime bin edges come from the background data of the explainer, not from the model. An attacker can make them from its own data.

The numbers

Tests used six datasets and five target families. Ranking gave no systematic increase. Bin edges gave a maximum of 41 points. Cutpurse beat the undefended published attack by 7.0 points.

What is not known

The results are for default Lime and Shap setups on tabular classifiers only. They do not show the same for different explainers, different targets or attackers with no data.

Insight: One hidden field is a weak defence

Here the defender hides bin edges, and the attacker makes them again from its own data. One earlier paper says router logs help an attack find 2.7 to 9.4 percentage points more examples of tuning. Side outputs that look harmless, such as explanations and logs, seem to leak more than owners think. Count each side output that your model service gives, not only the answer.