How to fix: unable to deserialize data after adding fields

As a beginner in android, I made a rookie mistake. I used JSON string to store some data and I used Proguard as default to publish my app Tk Rally Tennis scorekeeper. Well, I knew nothing about proguard. It was just one more file in my app I don’t know anything about. The thing is that this file makes all your variables in a published app looks like one or two letters. So my winnerPlayer1 variable becomes A or CD or something. This is for security reason so others will get really hard to understand your code. I think it boosts performance to…  

So what happens when I added more variables to my UndoRedo class. Well at beginning nothing. Everything seems to work just fine. But when i published my app my users told me that they can’t see old saved matches and I have a lot of crashes.
My old matches data was corrupted.  

I was not able to find any quick fix for this but if someone knows it please leave a comment. So my fix to this was like some detective work. First i figured out how was my data saved in published app and i did that by adding a button in my app that sends my data via email. Than i get my JSON data from unpublished app (the one in debugging mode).

This is how my published part of JSON data look like in published app

[{“A”:0,”B”:0,”C”:0,”D”:0,”E”:0,”F”:0,”G”:0,”H”:0,”I”:0,”J”:0,”K”:0,”L”:0,”M”:0,”N”:0,”O”:0,”P”:0,”Q”:0,”R”:0,”S”:0,”T”:0,”U”:0,”V”:0,”W”:0,”X”:0,”Y”:0,”Z”:0,”a”:0,”aa”:0,”ab”:0,”ac”:0,”ad”:0,”ae”:0,”af”:0,”ag”:0,”ah”:0,”ai”:0,”aj”:0,”ak”:0,”al”:0,”am”:”0″,”an”:”0″,”ao”:”0″,”ap”:”0″,”aq”:”0″,”ar”:”0″,”as”:””,”at”:0,”b”:0,”c”:0,”d”:0,”e”:0,”f”:0,”g”:[0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],”h”:2,”i”:0,”j”:true,”k”:true,”l”:false,”m”:true,”n”:false,”o”:true,”p”:0,”q”:0,”r”:0,”s”:0,”t”:0,”u”:0,”v”:0,”w”:0,”x”:0,”y”:0,”z”:0},{“A”:0,”B”:0,”C”:1,”D”:0,”E”:0,”F”:0,”G”:0,”H”:0,”I”:0,”J”:0,”K”:0,”L”:0,”M”:0,”N”:0,”O”:0,”P”:0,”Q”:0,”R”:0,”S”:0,”T”:0,”U”:0,”V”:0,”W”:0,”X”:0,”Y”:0,”Z”:0,”a”:0,”aa”:0,”ab”:0,”ac”:0,”ad”:0,”ae”:0,”af”:0,”ag”:0,”ah”:0,”ai”:0,”aj”:0,”ak”:0,”al”:0,”am”:”0″,”an”:”0″,”ao”:”0″,”ap”:”0″,”aq”:”0″,”ar”:”0″,”as”:””,”at”:-11059,”b”:0,”c”:0,”d”:0,”e”:0,”f”:0,”g”:[0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],”h”:2,”i”:0,”j”:false,”k”:true,”l”:false,”m”:false,”n”:false,”o”:true,”p”:0,”q”:0,”r”:0,”s”:0,”t”:0,”u”:0,”v”:0,”w”:0,”x”:1,”y”:0,”z”:0},{“A”:0,”B”:0,”C”:1,”D”:0,”E”:1,”F”:0,”G”:0,”H”:0,”I”:0,”J”:0,”K”:1,”L”:0,”M”:0,”N”:0,”O”:0,”P”:0,”Q”:1,”R”:0,”S”:0,”T”:0,”U”:0,”V”:0,”W”:0,”X”:0,”Y”:0,”Z”:0,”a”:15,”aa”:0,”ab”:0,”ac”:0,”ad”:0,”ae”:0,”af”:0,”ag”:0,”ah”:0,”ai”:0,”aj”:0,”ak”:0,”al”:0,”am”:”15″,”an”:”0″,”ao”:”0″,”ap”:”0″,”aq”:”0″,”ar”:”0″,”as”:”Point for Player 1″,”at”:-12611,”b”:0,”c”:0,”d”:0,”e”:0,”f”:0,”g”:[0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],”h”:2,”i”:0,”j”:false,”k”:true,”l”:false,”m”:true,”n”:false,”o”:true,”p”:1,”q”:0,”r”:0,”s”:0,”t”:0,”u”:0,”v”:0,”w”:0,”x”:1,”y”:0,”z”:0},{“A”:0,”B”:0,”C”:2,”D”:0,”E”:1,”F”:0,”G”:0,”H”:0,”I”:0,”J”:0,”K”:1,”L”:0,”M”:0,”N”:0,”O”:0,”P”:0,”Q”:1,”R”:0,”S”:0,”T”:0,”U”:0,”V”:0,”W”:0,”X”:0,”Y”:0,”Z”:0,”a”:15,”aa”:0,”ab”:0,”ac”:0,”ad”:0,”ae”:0,”af”:0,”ag”:0,”ah”:0,”ai”:0,”aj”:0,”ak”:0,”al”:0,”am”:”15″,”an”:”0″,”ao”:”0″,”ap”:”0″,”aq”:”0″,”ar”:”0″,”as”:”Point for Player 1″,”at”:-15617,”b”:0,”c”:0,”d”:0,”e”:0,”f”:0,”g”:[0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],”h”:2,”i”:0,”j”:false,”k”:true,”l”:false,”m”:false,”n”:false,”o”:true,”p”:1,”q”:0,”r”:0,”s”:0,”t”:0,”u”:0,”v”:0,”w”:0,”x”:2,”y”:0,”z”:0},{“A”:0,”B”:0,”C”:2,”D”:0,”E”:2,”F”:0,”G”:0,”H”:0,”I”:0,”J”:0,”K”:1,”L”:1,”M”:0,”N”:0,”O”:0,”P”:1,”Q”:1,”R”:0,”S”:0,”T”:0,”U”:0,”V”:0,”W”:0,”X”:0,”Y”:0,”Z”:0,”a”:15,”aa”:0,”ab”:0,”ac”:0,”ad”:0,”ae”:0,”af”:0,”ag”:0,”ah”:0,”ai”:0,”aj”:0,”ak”:0,”al”:0,”am”:”15″,”an”:”0″,”ao”:”0″,”ap”:”15″,”aq”:”0″,”ar”:”0″,”as”:”Point for Player 2″,”at”:-16896,”b”:15,”c”:0,”d”:0,”e”:0,”f”:0,”g”:[0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],”h”:2,”i”:0,”j”:false,”k”:true,”l”:false,”m”:true,”n”:false,”o”:true,”p”:1,”q”:0,”r”:0,”s”:0,”t”:0,”u”:0,”v”:1,”w”:0,”x”:2,”y”:0,”z”:0}]

And this is part of JSON in debugging mode

[{“mAcePlayer1″:0,”mAcePlayer2″:0,”mBreakPointsPlayer1″:0,”mBreakPointsPlayer2″:0,”mBreakPointsWonPlayer1″:0,”mBreakPointsWonPlayer2″:0,”mChronometerTime”:-1469,”mDisplayGamesPlayer1″:”0″,”mDisplayGamesPlayer2″:”0″,”mDisplayPointsPlayer1″:”0″,”mDisplayPointsPlayer2″:”0″,”mDisplaySetsPlayer1″:”0″,”mDisplaySetsPlayer2″:”0″,”mDisplayTextMessage”:”Discover the Beauty of the Game”,”mDoubleFaultPlayer1″:0,”mDoubleFaultPlayer2″:0,”mFaultPlayer1″:0,”mFaultPlayer2″:0,”mFirstFault”:true,”mFirstServePointsWonPlayer1″:0,”mFirstServePointsWonPlayer2″:0,”mFirstServicePlayer1″:0,”mFirstServicePlayer2″:0,”mForcedErrorPlayer1″:0,”mForcedErrorPlayer2″:0,”mGamesPlayer1″:0,”mGamesPlayer2″:0,”mMatchWon”:false,”mNumberOfBreakPoints”:0,”mNumberOfServeInTieBreak”:0,”mNumberOfSetsForWin”:2,”mPlayer1TypeBackhandError”:0,”mPlayer1TypeBackhandVolleyError”:0,”mPlayer1TypeBackhandVolleyWinner”:0,”mPlayer1TypeBackhandWinner”:0,”mPlayer1TypeForehandError”:0,”mPlayer1TypeForehandVolleyError”:0,”mPlayer1TypeForehandVolleyWinner”:0,”mPlayer1TypeForehandWinner”:0,”mPlayer1TypeSmashError”:0,”mPlayer1TypeSmashWinner”:0,”mPlayer2TypeBackhandError”:0,”mPlayer2TypeBackhandVolleyError”:0,”mPlayer2TypeBackhandVolleyWinner”:0,”mPlayer2TypeBackhandWinner”:0,”mPlayer2TypeForehandError”:0,”mPlayer2TypeForehandVolleyError”:0,”mPlayer2TypeForehandVolleyWinner”:0,”mPlayer2TypeForehandWinner”:0,”mPlayer2TypeSmashError”:0,”mPlayer2TypeSmashWinner”:0,”mPointsPlayer1″:0,”mPointsPlayer2″:0,”mPointsWonPlayer1″:0,”mPointsWonPlayer2″:0,”mReceiverPointsWonPlayer1″:0,”mReceiverPointsWonPlayer2″:0,”mSecondServePointsWonPlayer1″:0,”mSecondServePointsWonPlayer2″:0,”mSecondServicePlayer1″:0,”mSecondServicePlayer2″:0,”mServeOfPlayer”:true,”mServeOfPlayerInTieBreak”:true,”mSetsPlayer1″:0,”mSetsPlayer2″:0,”mSetsScore”:[0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],”mTieBreak”:false,”mTiebreakFinal”:true,”mUnforcedErrorPlayer1″:0,”mUnforcedErrorPlayer2″:0,”mWinnerPlayer1″:0,”mWinnerPlayer2″:0},

Detective work

Then I figure out what is A and B and C really is and make a list of it. This was way more difficult than I thought it would be. Because variables are not in the same order. So I made lots of different JSONs with just a few variables that change and then compare them.  This took me a lot of time because there is no room for mistake. Then I created a method that gets old JSON data and converts it to new JSON data. My method looks like this.  

public static String convertOldMatchGsonToNew(String oldGson) {
String convertedGson = oldGson;
String[][] replacements = {
{"\"A\":", "\"mUnforcedErrorPlayer2\":"},
{"\"B\":", "\"mFirstServicePlayer1\":"},
{"\"C\":", "\"mFirstServicePlayer2\":"},
{"\"D\":", "\"mSecondServicePlayer1\":"},
{"\"E\":", "\"mSecondServicePlayer2\":"},
{"\"F\":", "\"mBreakPointsPlayer1\":"},
{"\"G\":", "\"mBreakPointsPlayer2\":"},
{"\"H\":", "\"mBreakPointsWonPlayer1\":"},
{"\"I\":", "\"mBreakPointsWonPlayer2\":"},
{"\"J\":", "\"mNumberOfBreakPoints\":"},
{"\"K\":", "\"mPointsWonPlayer1\":"}, //test it
{"\"L\":", "\"mPointsWonPlayer2\":"}, // test it
{"\"M\":", "\"mFirstServePointsWonPlayer1\":"},
{"\"N\":", "\"mFirstServePointsWonPlayer2\":"},
{"\"O\":", "\"mSecondServePointsWonPlayer1\":"},
{"\"P\":", "\"mSecondServePointsWonPlayer2\":"},
{"\"Q\":", "\"mReceiverPointsWonPlayer1\":"},
{"\"R\":", "\"mReceiverPointsWonPlayer2\":"},
{"\"S\":", "\"mPlayer1TypeBackhandWinner\":"},
{"\"T\":", "\"mPlayer1TypeBackhandVolleyWinner\":"},
{"\"U\":", "\"mPlayer1TypeForehandWinner\":"},
{"\"V\":", "\"mPlayer1TypeForehandVolleyWinner\":"},
{"\"W\":", "\"mPlayer1TypeSmashWinner\":"},
{"\"X\":", "\"mPlayer2TypeBackhandWinner\":"},
{"\"Y\":", "\"mPlayer2TypeBackhandVolleyWinner\":"},
{"\"Z\":", "\"mPlayer2TypeForehandWinner\":"},
{"\"a\":", "\"mPointsPlayer1\":"},
{"\"aa\":", "\"mPlayer2TypeForehandVolleyWinner\":"},
{"\"ab\":", "\"mPlayer2TypeSmashWinner\":"},
{"\"ac\":", "\"mPlayer1TypeBackhandError\":"},
{"\"ad\":", "\"mPlayer1TypeBackhandVolleyError\":"},
{"\"ae\":", "\"mPlayer1TypeForehandError\":"},
{"\"af\":", "\"mPlayer1TypeForehandVolleyError\":"},
{"\"ag\":", "\"mPlayer1TypeSmashError\":"},
{"\"ah\":", "\"mPlayer2TypeBackhandError\":"},
{"\"ai\":", "\"mPlayer2TypeBackhandVolleyError\":"},
{"\"aj\":", "\"mPlayer2TypeForehandError\":"},
{"\"ak\":", "\"mPlayer2TypeForehandVolleyError\":"},
{"\"al\":", "\"mPlayer2TypeSmashError\":"},
{"\"am\":", "\"mDisplayPointsPlayer1\":"},
{"\"an\":", "\"mDisplayGamesPlayer1\":"},
{"\"ao\":", "\"mDisplaySetsPlayer1\":"},
{"\"ap\":", "\"mDisplayPointsPlayer2\":"},
{"\"aq\":", "\"mDisplayGamesPlayer2\":"},
{"\"ar\":", "\"mDisplaySetsPlayer2\":"},
{"\"as\":", "\"mDisplayTextMessage\":"},
{"\"at\":", "\"mChronometerTime\":"},
{"\"b\":", "\"mPointsPlayer2\":"},
{"\"c\":", "\"mGamesPlayer1\":"},
{"\"d\":", "\"mGamesPlayer2\":"},
{"\"e\":", "\"mSetsPlayer1\":"},
{"\"f\":", "\"mSetsPlayer2\":"},
{"\"g\":", "\"mSetsScore\":"},
{"\"h\":", "\"mNumberOfSetsForWin\":"},
{"\"i\":", "\"mNumberOfServeInTieBreak\":"},
{"\"j\":", "\"mServeOfPlayer\":"},
{"\"k\":", "\"mServeOfPlayerInTieBreak\":"},
{"\"l\":", "\"mTieBreak\":"},
{"\"m\":", "\"mFirstFault\":"},
{"\"n\":", "\"mMatchWon\":"},
{"\"o\":", "\"mTiebreakFinal\":"},
{"\"p\":", "\"mWinnerPlayer1\":"},
{"\"q\":", "\"mAcePlayer1\":"},
{"\"r\":", "\"mFaultPlayer1\":"},
{"\"s\":", "\"mDoubleFaultPlayer1\":"},
{"\"t\":", "\"mForcedErrorPlayer1\":"},
{"\"u\":", "\"mUnforcedErrorPlayer1\":"},
{"\"w\":", "\"mWinnerPlayer2\":"},
{"\"v\":", "\"mAcePlayer2\":"},
{"\"x\":", "\"mFaultPlayer2\":"},
{"\"y\":", "\"mDoubleFaultPlayer2\":"},
{"\"z\":", "\"mForcedErrorPlayer2\":"},
};


for (int i = 0; i < replacements.length; i++) {
convertedGson = convertedGson.replace(replacements[i][0], replacements[i][1]);
}
return convertedGson;
}

Like you can see this is just a list of strings array that goes thru loop that replaces all A, B, C and so on with the corresponding variable name. You just need to make your own replacement list that is unique. (This can be really hard, if u have like more than 100 variables you are in big trouble. You may need days for making this list.)

Next thing I did is to put this method when i load my data so in my case it was in my cursor adapter and my catalog activity and my implementation look like this:

switch (item.getItemId()) {
case R.id.action_match_load:
String gsonString = MatchCatalogActivity.matchInformation.get(indexMatchInfo).getMatchArrayList();
//convert Gson if it is old version
if (gsonString.contains("\"A\":")){
String convertedGson = Utility.convertOldMatchGsonToNew(gsonString);
NavigationScoreKeeperActivity.savedState = Utility.getUndoRedoArrayListFromString(convertedGson);
}else {
NavigationScoreKeeperActivity.savedState = Utility.getUndoRedoArrayListFromString(gsonString);
}

and this is in my catalog activity:

//convert gson if it is old version
if (gsonString.contains("\"A\":")) {
gsonString = Utility.convertOldMatchGsonToNew(gsonString);
ContentValues values = new ContentValues();
values.put(MatchEntry.COLUMN_MATCH_ARRAY_LIST, gsonString);
Uri mCurrentMatchUri = ContentUris.withAppendedId(PlayerContract.MatchEntry.CONTENT_URI, matchID);
getContentResolver().update(mCurrentMatchUri, values, null, null);
}

I made a simple check for my “corrupted” data if (gsonString.contains(“\”A\”:”)) that looks for changed variable name inside JSON data that is always appear in every “corrupted” JSON string. U may need to look for some other variable name.

Most important part

Don’t forget to change your proguard-rules.pro file so you wont be in same problem again, there is lots of documentation on web how to do so. Here is how my
proguard-rules.pro file looks now:

# You can edit the include path and order by changing the proguardFiles
# directive in build.gradle.
#
# For more details, see
# http://developer.android.com/guide/developing/tools/proguard.html

# Add any project specific keep options here:

# If your project uses WebView with JS, uncomment the following
# and specify the fully qualified class name to the JavaScript interface
# class:
#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
# public *;
#}

-dontwarn com.fangxu.**
##---------------Begin: proguard configuration for Gson ----------
# Gson uses generic type information stored in a class file when working with fields. Proguard
# removes such information by default, so configure it to keep all of it.
-keepattributes Signature

# For using GSON @Expose annotation
-keepattributes *Annotation*

# Gson specific classes
-dontwarn sun.misc.**
#-keep class com.google.gson.stream.** { *; }

# Application classes that will be serialized/deserialized over Gson
-keep class com.google.gson.examples.android.model.** { *; }

# Prevent proguard from stripping interface information from TypeAdapterFactory,
# JsonSerializer, JsonDeserializer instances (so they can be used in @JsonAdapter)
-keep class * implements com.google.gson.TypeAdapterFactory
-keep class * implements com.google.gson.JsonSerializer
-keep class * implements com.google.gson.JsonDeserializer

-keep class com.akristic.www.tkrally.UndoRedo

Hope this can help someone.