Why Backslash character not removed even after using replace method

Question

I have made a rest request which is returning me a Set in JSON format which is "[\"TestBack\"]".

If I directly parse it in Apex using

Set<String> rw = (Set<String>)JSON.deserialize(response.getBody(),Set<String>.class);

then I get following error

FATAL_ERROR System.JSONException: Malformed JSON: Expected '[' at the beginning of List/Set

but if I explicitly remove double quote sign by using

 Set<String> rw = (Set<String>)JSON.deserialize(response.getBody().substringAfter('"').substringBeforeLast('"'),Set<String>.class);
                      

I got following error

FATAL_ERROR System.JSONException: Unexpected character ('\' (code 92)): expected a valid value 

and if I try to use replaceAll method

Set<String> rw = (Set<String>)JSON.deserialize(response.getBody().substringAfter('"').substringBeforeLast('"').replaceAll('\\',''),Set<String>.class);

it shows following error

FATAL_ERROR System.StringException: Invalid regex: Unexpected internal error near index 1                

Is there any way to get parse back into Set?