Request for Comments: Use default_charset As Default Character Encoding
This RFC proposes that use default_charset as default character encoding.
Current PHP does not have default encoding setting. This makes adoption of PHP 5.4 difficult, since PHP 5.4’s htmlentities/htmlspecialchars is now default to UTF-8. Some applications are required to set proper encoding for htmlentities/htmlspecialchars for proper character processing. If users mixed ISO-8859-1 and UTF-8 (AND many other multibyte character encodings), it could cause security problem.
There are many encoding setting in php.ini and functions that users simply ignore and leave it alone. However, it is required to handle character encoding properly for secure programs.
Objectives of this proposal are:
Setting charset in HTTP header is recommended since the first XSS advisory in 2000 Feb. by CERT and Microsoft. (Better security)
If we have yet another multibyte string module in the future, the new common ini settings can be used. (No more module specific INIs)
Proposal
Set default_charset=“UTF-8” as PHP default for both compiled and php.ini-* option.
Add php.input_encoding, php.internal_encoding and php.output_encoding for encoding related module/functions.
Use default_charset as default for encoding related php.ini settings and module/functions.
PHP 5.6 and master, introduce new php.ini setting. Old iconv.*/mbstring.* php.ini parameters will be removed for master PHP6. Use of iconv.*/mbstring.* php.ini parameters raise E_DEPRECATED for 5.6 and up.
all functions that take encoding option use php.internal_encoding as default (e.g. htmlentities/mb_strlen/mb_regex/etc)
Precedence of settings
Encoding name handling
mbstring and iconv have different level of support.
iconv does not have API for getting supported encoding and iconv is built with system’s iconv library.
users are responsible to set proper encoding name. e.g. mbstring has SJIS-win, but iconv only has SJIS
Use cases
It simplify i18n applications.
Unifies *.output_encoding/*.internal_encoding/*.input_encoding setting.
Users may check default_charset see if encoding conversion is needed or not. For example, pcre/sqlite only suports UTF-8 and users may check & convert encoding as follows.
if (ini_get('default_charset') !== 'UTF-8') < $str = mb_convert_encoding($str, 'UTF-8'); >preg, sqlite function calls here. Other related issues
escapeshellcmd/escapeshellarg/fgetcsv or like, are using locale based MBCS support via php_mblen(). These functions are out side of this RFC scope.
Database character encoding is also out side of this RFC scope.
BC issues
None when users already using UTF-8 as their encoding.
Other users may have to change “default_encoding” php.ini setting (leave it empty or set it to desired encoding)
Php ini set encoding
The behaviour of these functions is affected by settings in php.ini .
| Name | Default | Changeable | Changelog |
|---|---|---|---|
| mbstring.language | «neutral» | PHP_INI_ALL | |
| mbstring.detect_order | NULL | PHP_INI_ALL | |
| mbstring.http_input | «pass» | PHP_INI_ALL | Deprecated |
| mbstring.http_output | «pass» | PHP_INI_ALL | Deprecated |
| mbstring.internal_encoding | NULL | PHP_INI_ALL | Deprecated |
| mbstring.substitute_character | NULL | PHP_INI_ALL | |
| mbstring.func_overload | «0» | PHP_INI_SYSTEM | Deprecated as of PHP 7.2.0; removed as of PHP 8.0.0. |
| mbstring.encoding_translation | «0» | PHP_INI_PERDIR | |
| mbstring.http_output_conv_mimetypes | «^(text/|application/xhtml\+xml)» | PHP_INI_ALL | |
| mbstring.strict_detection | «0» | PHP_INI_ALL | |
| mbstring.regex_retry_limit | «1000000» | PHP_INI_ALL | Available as of PHP 7.4.0. |
| mbstring.regex_stack_limit | «100000» | PHP_INI_ALL | Available as of PHP 7.3.5. |
For further details and definitions of the PHP_INI_* modes, see the Where a configuration setting may be set.
Here’s a short explanation of the configuration directives.
The default national language setting (NLS) used in mbstring. Note that this option automagically defines mbstring.internal_encoding and mbstring.internal_encoding should be placed after mbstring.language in php.ini
Enables the transparent character encoding filter for the incoming HTTP queries, which performs detection and conversion of the input encoding to the internal character encoding.
This deprecated feature will certainly be removed in the future.
Defines the default internal character encoding.
Users should leave this empty and set default_charset instead.
This deprecated feature will certainly be removed in the future.
Defines the default HTTP input character encoding.
Users should leave this empty and set default_charset instead.
This deprecated feature will certainly be removed in the future.
Defines the default HTTP output character encoding (output will be converted from the internal encoding to the HTTP output encoding upon output).
Users should leave this empty and set default_charset instead.
Defines default character code detection order. See also mb_detect_order() .
Defines character to substitute for invalid character encoding. See mb_substitute_character() for supported values.
This feature has been DEPRECATED as of PHP 7.2.0, and REMOVED as of PHP 8.0.0. Relying on this feature is highly discouraged.
Overloads a set of single byte functions by the mbstring counterparts. See Function overloading for more information.
This setting can only be changed from the php.ini file.
Enables strict encoding detection. See mb_detect_encoding() for a description and examples.
Limits the amount of backtracking that may be performed during one mbregex match.
This setting only takes effect when linking against oniguruma >= 6.8.0.
Limits the stack depth of mbstring regular expressions.
According to the » HTML 4.01 specification, Web browsers are allowed to encode a form being submitted with a character encoding different from the one used for the page. See mb_http_input() to detect character encoding used by browsers.
Although popular browsers are capable of giving a reasonably accurate guess to the character encoding of a given HTML document, it would be better to set the charset parameter in the Content-Type HTTP header to the appropriate value by header() or default_charset ini setting.
Example #1 php.ini setting examples
; Set default language mbstring.language = Neutral; Set default language to Neutral(UTF-8) (default) mbstring.language = English; Set default language to English mbstring.language = Japanese; Set default language to Japanese ;; Set default internal encoding ;; Note: Make sure to use character encoding works with PHP mbstring.internal_encoding = UTF-8 ; Set internal encoding to UTF-8 ;; HTTP input encoding translation is enabled. mbstring.encoding_translation = On ;; Set default HTTP input character encoding ;; Note: Script cannot change http_input setting. mbstring.http_input = pass ; No conversion. mbstring.http_input = auto ; Set HTTP input to auto ; "auto" is expanded according to mbstring.language mbstring.http_input = SJIS ; Set HTTP input to SJIS mbstring.http_input = UTF-8,SJIS,EUC-JP ; Specify order ;; Set default HTTP output character encoding mbstring.http_output = pass ; No conversion mbstring.http_output = UTF-8 ; Set HTTP output encoding to UTF-8 ;; Set default character encoding detection order mbstring.detect_order = auto ; Set detect order to auto mbstring.detect_order = ASCII,JIS,UTF-8,SJIS,EUC-JP ; Specify order ;; Set default substitute character mbstring.substitute_character = 12307 ; Specify Unicode value mbstring.substitute_character = none ; Do not print character mbstring.substitute_character = long ; Long Example: U+3000,JIS+7E7E
Example #2 php.ini setting for EUC-JP users
;; Disable Output Buffering output_buffering = Off ;; Set HTTP header charset default_charset = EUC-JP ;; Set default language to Japanese mbstring.language = Japanese ;; HTTP input encoding translation is enabled. mbstring.encoding_translation = On ;; Set HTTP input encoding conversion to auto mbstring.http_input = auto ;; Convert HTTP output to EUC-JP mbstring.http_output = EUC-JP ;; Set internal encoding to EUC-JP mbstring.internal_encoding = EUC-JP ;; Do not print invalid characters mbstring.substitute_character = none
Example #3 php.ini setting for SJIS users
;; Enable Output Buffering output_buffering = On ;; Set mb_output_handler to enable output conversion output_handler = mb_output_handler ;; Set HTTP header charset default_charset = Shift_JIS ;; Set default language to Japanese mbstring.language = Japanese ;; Set http input encoding conversion to auto mbstring.http_input = auto ;; Convert to SJIS mbstring.http_output = SJIS ;; Set internal encoding to EUC-JP mbstring.internal_encoding = EUC-JP ;; Do not print invalid characters mbstring.substitute_character = none
User Contributed Notes 3 notes
String literals in the PHP script are encoded with the same encoding that the PHP file was saved with. This is not affected by default_charset or other .ini settings.
Scenario: The default_charset is KOI8-R, and there is a text file «input.txt» containing the string «Это текст для поиска.» in KOI8-R encoding.
$data = file_get_contents ( ‘input.txt’ );
echo mb_strpos ( $data , $string );
?>
But unfortunately it was saved as UTF-8.
It doesn’t work; mb_strpos() returns false because it can’t find the UTF-8-encoded «текст» inside the KOI8-R-encoded «Это текст для поиска.».
Adjusting the default_charset had no effect. Not even fiddling with mb_internal_encoding could fix it, simply because the strings involved had *different* encodings and without actually changing one of them they just weren’t going to match.
Either re-save the source file as KOI8-R to match the data file, or re-save the data file as UTF-8 to match the source code. Only then will the script properly echo ‘4’.
The documentation is vague, on WHAT precisely the valid «NLS» language strings are that are valid for «mbstring.language».
According to http://php.net/manual/en/function.mb-language.php the values are «Japanese», «ja», «English», «en», or «uni» for UTF-8.
On the other hand, the sample on this current page omits «uni» but introduces «Neutral» as an undocumented option — which is also the default value:
var_dump ( mb_language () ); // «neutral» (default if not set)
var_dump ( mb_language ( ‘uni’ ) ); // TRUE, valid language string
var_dump ( mb_language () ); // «uni»
var_dump ( mb_language ( ‘neutral’ ) ); // TRUE, valid language string
var_dump ( mb_language () ); // «neutral»
?>
Note that you should better at least set «mbstring.internal_encoding».
echo mb_internal_encoding () . ‘
‘ ;
echo mb_regex_encoding ();
?>
You might be surprised at unexpected values.
mbstring.language Japanese
;mbstring.internal_encoding (commented out showing «no value» in phpinfo() )
These two lines in «php.ini» are the same values as
«mbstring.internal_encoding» defines the default encoding for «mb_» Functions such as «mb_strlen()».
It also defines the same for «mb_ereg_» Functions such as «mb_ereg()» when you don’t set «mb_regex_encoding».