Имитация браузера через curl
Дайте подсказку. Делаю парсер. Для парсинга мне необходимо авторизоваться на сайте. Но сайт палит, что отправка логина и пароля не с браузера. Вот как выглядит curl :
$this->ch = curl_init(); curl_setopt($this->ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"); curl_setopt($this->ch, CURLOPT_SSL_VERIFYHOST, false); curl_setopt($this->ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($this->ch, CURLOPT_COOKIEFILE, 'user_cookie_file.txt'); curl_setopt($this->ch, CURLOPT_COOKIEJAR, 'user_cookie_file.txt'); curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, true); curl_setopt($this->ch, CURLOPT_URL, PAGEAUTHORIZATION); curl_setopt($this->ch, CURLOPT_POST, 1); curl_setopt($this->ch, CURLOPT_POSTFIELDS, $this->postData); curl_setopt($this->ch, CURLOPT_REFERER, PAGEAUTHORIZATION); $error = curl_exec($this->ch); Параметры в пост запросе отправляю все что нужно. Авторизация иногда проходит, пример 1 раз из 20 попыток. В остальных случаях в отчет получаю страницу сайта с предупреждением о подозрительной активности. Прошу дать совет.
1 ответ 1
Попробуй посмотреть заголовки, которые ты отправляешь когда авторизовываешься, используя браузер, а затем эти заголовки подставить в свой запрос через curl
curl_setopt ($ch, CURLOPT_HTTPHEADER, $headers);
Тут от конкретного сайта зависит, кто что придумывает, чтобы обезопасить себя и данные
Это сработало, но не надолго. Там есть заголовок, который отправляет браузер «Cookie:» там данных капец сколько. Походу по ним как то сайт понимает спустя время что я робот. Не подскажите по каким именно критериям он может понимать? Список кук там просто безразмерный и 80% нечитаемы.
Для кук используется CURLOPT_COOKIEFILE CURLOPT_COOKIEJAR и они уже есть. Можно попробовать еще такой вариант — не сразу отправлять этот запрос curl а перед ним отправить просто GET запрос на страницу авторизации, т.е. симулировать действия пользователя «Заход на страницу» -> «Отправка формы» Иногда разработчики делают так, что при входе на страницу авторизации, сразу навешивают куки, которые нужно потом использоваться при отправке POST данных
Php curl emulate a browser?
problem solved, now the headers is even in the correct order, and the request seems to be COMPLETELY INDISTINGUISHABLE from the real firefox request 🙂 (i don’t actually recommend this last step, it’s a maintenance burden to keep CURLOPT_ENCODING in sync with the custom Accept-Encoding header, and i’ve never experienced a situation where the order of the headers are significant),and voila! our php-emulated browser GET request should now be indistinguishable from the real firefox GET request :), How to keep pee from splattering from the toilet all around the basin and on the floor on old toilets that are really low and have deep water? ,i’ll make an example, first decide what browser you want to emulate, in this case i chose Firefox 60.6.1esr (64-bit), and check what GET request it issues, this can be obtained with a simple netcat server (MacOS bundles netcat, most linux distributions bunles netcat, and Windows users can get netcat from.. Cygwin.org , among other places),
edit: Since the request uses https there might also be error in verifying the certificate, see CURLOPT_SSL_VERIFYPEER.
$url="https://new.aol.com/productsweb/subflows/ScreenNameFlow/AjaxSNAction.do?s=username&f=firstname&l=lastname"; $agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)'; $ch = curl_init(); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($ch, CURLOPT_VERBOSE, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_USERAGENT, $agent); curl_setopt($ch, CURLOPT_URL,$url); $result=curl_exec($ch); var_dump($result); Answer by Celine Fuller
PHP USES curl to simulate user login code , php uses curl to grab examples of visitor information in qq space , php USES curl to simulate the collection of pages after login , php uses curl to simulate the method of uploading files or pictures from browser forms
The official explanation
curl is a file transfer tool that USES the URL syntax to work from the command line. curl is a file transfer tool that USES the URL syntax to work from the command line.
It supports many protocols: FTP, FTPS, HTTP, HTTPS, GOPHER, TELNET, DICT, FILE and LDAP. curl also supports HTTPS authentication, HTTP POST method, HTTP PUT method, FTP upload, kerberos authentication, HTTP upload, proxy server, cookies, user name/password authentication, download file breakpoint and continue,
Upload file breakpoint continuation, http proxy server pipeline (proxy tunneling), it even supports IPv6, socks5 proxy server, upload files to FTP server through http proxy server, etc., with 10 powerful functions.
The curl function is applied in PHP
Simply put, there are 4 steps to 1
curl_init();
curl_setopt();
curl_exec();
curl_close();
The most important command is curl_setopt();
1 simple post request example
index.php
Answer by Alina Stanton
Installing/ConfiguringRequirementsInstallationRuntime ConfigurationResource Types,curl_error — Return a string containing the last error for the current session,curl_multi_info_read — Get information about the current transfers,curl_multi_exec — Run the sub-connections of the current cURL handle
I wrote the following to see if a submitted URL has a valid http response code and also if it responds quickly. Use the code like this: The second argument is optional, and it allows you to check for a specific response code The third allows you to specify how long you are willing to wait for a response. else if ($pid) < // we are the parent $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_HEADER, TRUE); curl_setopt($ch, CURLOPT_NOBODY, TRUE); // remove body curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); $head = curl_exec($ch); $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE); curl_close($ch); if(!$head) < return FALSE; >if($status === null) < if($httpCode < 400) < return TRUE; >else < return FALSE; >> elseif($status == $httpCode) < return TRUE; >return FALSE; pcntl_wait($status); //Protect against Zombie children > else < // we are the child while(microtime(true) < $expire) < sleep(0.5); >return FALSE; > > ?> Hope this example helps. It is not 100% tested, so any feedback [sent directly to me by email] is appreciated. Answer by Abner Phillips
5Whether a qualitative self-study of math?,1How do I output the response from the two ajax requests in one script?,1How to convert SVG to PSD, but keeping the elements on separate layers?,Find more questions by tags PHPWeb Development
'https://ru.aliexpress.com/category/202003409/tops-tees.html?site=rus&g=y&SortType=total_tranpro_desc&needQuery=n&tag=', CURLOPT_COOKIE => ", CURLOPT_RETURNTRANSFER => true, CURLOPT_HEADER => 0, CURLOPT_COOKIESESSION => true, CURLOPT_COOKIEFILE => $_SERVER['DOCUMENT_ROOT'].'/cookie.txt', CURLOPT_REFERER => null, CURLOPT_USERAGENT => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", CURLOPT_TIMEOUT => 10, CURLOPT_RETURNTRANSFER => 1, CURLOPT_FOLLOWLOCATION => 1, ); curl_setopt_array($cs, $opt); $out = curl_exec($cs); echo $out; curl_close($cs); ?>Answer by Israel Chen
This is a very basic thing to do with cURL, but with endless possibilities. Once you have a webpage in a PHP variable, you can for example, retrieve a particular information on the page to use on your own website.,Twitter is very popular since some time now, and you probably already have an account there. (We have one too) So, what about using cURL to tweet from your server without connectiong to Twitter?,New to cURL? If yes, check out the following articles to learn the purposes and basics of cURL/libcurl., 10 – Post to Twitter using PHP and cURL
Wanna update your facebook status, but don’t want to go to facebook.com, login, and finally being able to update your status? Simply save the following code on your server, define the variables, and voilà !
'.ucfirst($first_name).'/', $page, $form_id); curl_setopt($ch, CURLOPT_POSTFIELDS,'post_form_id='.$form_id[1].'&status='.urlencode($status).'&update=Update'); curl_setopt($ch, CURLOPT_URL, 'http://m.facebook.com/home.php'); curl_exec($ch); ?> Do you ever wanted to know the exact download speed of your webserver (or any other?) If yes, you’ll love that code. You just have to initialize the $url variable with any resources from the webserver (images, pdf, etc), place the file on your server and point your browser to it. The output will be a full report of download speed.
This function can post on your WordPress blog. You don’t need to login to your WP dashboard etc.
Though, you must activate the XMLRPC posting option in your WordPress blog. If this option isn’t activated, the code will not be able to insert anything into WordPress database. Another thing, make sure the XMLRPC functions are activated on your php.ini file.
function wpPostXMLRPC($title,$body,$rpcurl,$username,$password,$category,$keywords='',$encoding='UTF-8') < $title = htmlentities($title,ENT_NOQUOTES,$encoding); $keywords = htmlentities($keywords,ENT_NOQUOTES,$encoding); $content = array( 'title'=>$title, 'description'=>$body, 'mt_allow_comments'=>0, // 1 to allow comments 'mt_allow_pings'=>0, // 1 to allow trackbacks 'post_type'=>'post', 'mt_keywords'=>$keywords, 'categories'=>array($category) ); $params = array(0,$username,$password,$content,true); $request = xmlrpc_encode_request('metaWeblog.newPost',$params); $ch = curl_init(); curl_setopt($ch, CURLOPT_POSTFIELDS, $request); curl_setopt($ch, CURLOPT_URL, $rpcurl); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_TIMEOUT, 1); $results = curl_exec($ch); curl_close($ch); return $results; ?> I know, it sounds basic. In fact, it is basic, but it is also very useful, especially when you have to work with external resources.
In a previous article, I have discussed how spammers spams your WordPress blog. To do so, they simply have to fill the $postfields array with the info they want to display and load the page.
Of course, this code is only for educationnal purposes.
[email protected]"; $postfields["url"] = "http://www.iamaspammer.com/"; $postfields["comment"] = "I am a stupid spammer."; $postfields["comment_post_ID"] = "123"; $postfields["_wp_unfiltered_html_comment"] = "0d870b294b"; //Url of the form submission $url = "http://www.ablogthatdoesntexist.com/blog/suggerer_site.php?action=meta_pass&id_cat=0"; $useragent = "Mozilla/5.0"; $referer = $url; //Initialize CURL session $ch = curl_init($url); //CURL options curl_setopt($ch, CURLOPT_POST, 1); //We post $postfields data curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields); //We define an useragent (Mozilla/5.0) curl_setopt($ch, CURLOPT_USERAGENT, $useragent); //We define a refferer ($url) curl_setopt($ch, CURLOPT_REFERER, $referer); //We get the result page in a string curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //We exits CURL $result = curl_exec($ch); curl_close($ch); //Finally, we display the result echo $result; ?> If you’re a blogger, you’re probably using the popular FeedBurner service, which allo you to know how many people grabbed your rss feed. Feedburner have a chicklet to proudly display your subscriber count on your blog. I personally like the chicklet’s look, but I heard lots of bloggers complaining about it. happilly, cURL can simply grab the count value and return it to you as a variable so you can display it as you want on your blog.
//get cool feedburner count $whaturl="https://feedburner.google.com/api/awareness/1.0/GetFeedData?uri=feedburner-id"; //Initialize the Curl session $ch = curl_init(); //Set curl to return the data instead of printing it to the browser. curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Set the URL curl_setopt($ch, CURLOPT_URL, $whaturl); //Execute the fetch $data = curl_exec($ch); //Close the connection curl_close($ch); $xml = new SimpleXMLElement($data); $fb = $xml->feed->entry['circulation']; //end get cool feedburner count This is a very basic thing to do with cURL, but with endless possibilities. Once you have a webpage in a PHP variable, you can for example, retrieve a particular information on the page to use on your own website.
Twitter is very popular since some time now, and you probably already have an account there. (We have one too) So, what about using cURL to tweet from your server without connectiong to Twitter?
Answer by Makayla Erickson
When a URL is requested via cURL , the server responds to the request like a request sent via browser. However, the default cURL request do not specify information which is needed by some servers to validate a request, for eg. User Agent, language etc.,Now that we have set user agent for the cURL request, it’s time to set the header to make the request completely legit.,You can add as many user agents you want to and using the array_rand() function the strings will be picked randomly and sent to the server while making various cURL requests.,Now you are good to execute your cURL requests and have fun.
curl_setopt($curl,CURLOPT_USERAGENT,'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:7.0.1) Gecko/20100101 Firefox/7.0.1');