Plugins: How to set encoding for TStringList -> SaveToFile?

[pmk65]

Im having problems setting encoding for ANSI/UTF-8 documents when using TStringList -> SaveToFile

I tried various ways, but none of them worked.
(Like adding a 2nd parameter to SaveToFile or settting TStringList.DefaultEncoding / TStringList.Encoding.)

So how do I set the encoding for TStringList SaveToFile, so I can save in specific ANSI/UTF-8 formats?

Text in editor before I copy it to TStringList and save it using SaveToFile:

Code: Select all

.test {
	content: "bøllehår";
}

Then when I load the content back into TStringList using LoadFromFile and paste it back in editor, it looks like this:

Code: Select all

.test {
	content: "b�lleh�r";
}

Both extended characters ("ø" and "å") are turned into the SAME 3 character sequence ("�", hex: EF BF BD). So it's not possible to fix it "manually" either.

Doesn't matter if the content in editor is ANSI or UTF-8. Both produces the same buggy result.

[pmk65]

I tried using instead TFileStream instead, but I can only get it to write 1 char/byte to file no matter what I do.

My code:

Code: Select all

var len = Length(contents);
FS = new TFileStream(filename, fmCreate);
FS.Size = len;
FS.Write(contents, len);
delete FS;

[Aivars]

Don't set FS.Size, just use FS.Write. Since "contents" is UTF-16 string internally, you will need to use 2 x length to write it correctly, but it still won't be UTF8, which is probably what you need. I could add extra method(s) for TStrings to save in various encodings and/or export this function for pluginscripts: function SaveStringToFile(s, FName: string; Encoding: TEncoding): boolean;

[pmk65]

The TFileStream was just an experiment to see if that would help.
Thanks for the info about UTF-16. That explains a lot.

Adding an encoding method would really help. (In newer versions of Delphi, both "SaveToFile" and "LoadFromFile" has a 2nd parameter to set encoding. So there might be a need for a "Load" command with encoding too.)
It seems really stupid that they have implemented LoadFromFile and SaveToFile without a way to set the file encoding.

So "SaveStringToFile" would be a new command, where you don't have to setup a TStringList first?

Currently I "solved" the problem in my Beautifier and CSSComb plugins, by converting all Unicode chars to Entities before saving. (Not an optimal solution, but at least it works.)

[Aivars]

The problem is that FastScript does not support optional parameters, so it would have to be a new method, otherwise just adding the second parameter would break the old scripts.

Yes, SaveStringToFile will be a new global function, no need for any objects. I've already added this to my todo list.

[pmk65]

I have been experimenting a bit more with "CreateOleObject", and came up with these two functions:

Code: Select all

/**
 * Save data to file
 *
 * @param  string   file The path/name of file
 * @param  string   data The content to save
 * @param  int      enc The document encoding (Webuilder numeric format)
 *
 * @return void
 */
function SaveToFile(file, data, enc) {

    switch (enc) {
        case 0:   charset = "iso-8859-1";	// ANSI
        case 1,2: charset = "utf-8";		// UTF-8 & UTF-8 without BOM
        default:  charset = "utf-16";	// UTF-8
    }

    var stream = CreateOleObject("ADODB.Stream");
    stream.Open;
    stream.Position = 0;
    stream.CharSet = charset;
    stream.Type = 2;  				// 1 = binary, 2 = text
    stream.WriteText(data);
    stream.SaveToFile(file, 2); 	// 1 don't overwrite, 2= overwrite
    stream.Close;
}

/**
 * Load data to file
 *
 * @param  string   file The path/name of file
 * @param  int      enc The document encoding (Webuilder numeric format)
 *
 * @return string   the content of the file
 */
function FileGetContents(file, enc) {

    switch (enc) {
        case 0:   charset = "iso-8859-1";	// ANSI
        case 1,2: charset = "utf-8";		// UTF-8 & UTF-8 without BOM
        default:  charset = "utf-16";	// UTF-8
    }

    var stream = CreateOleObject("ADODB.Stream");
    stream.Open;
    stream.Position = 0;
    stream.CharSet = charset;
    stream.Type = 2;  				// 1 = binary, 2 = text
    stream.LoadFromFile(file);
    var data =stream.ReadText();
    stream.Close;
	return data;
}

With those, I can load/save the Editor content in ANSI, UTF-8 and UTF-16 formats.