-
Notifications
You must be signed in to change notification settings - Fork 768
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cURL error 35 #287
Comments
Hello @vovan47 It looks like the error message itself is cropped and only stack trace is shown. Best, |
Hi Knack! Sure, I just copypasted it wrongly and wasn't able to edit issue
after saving!
Error message is: ``` cURL error 35: error:02001018:system
library:fopen:Too many open files (see
http://curl.haxx.se/libcurl/c/libcurl-errors.html) ```
Also I need to mention that it happens during massive campaign creation
(like 500+ accounts at once) after some of them were processed succesfully
(~50).
Our guess is that might be some server limitations.
…On 16 May 2017 at 16:34, Thanet Knack Praneenararat < ***@***.***> wrote:
Hello @vovan47 <https://github.com/vovan47>
It looks like the error message itself is cropped and only stack trace is
shown.
Could you please re-produce the issue and fetch the error message as well?
Best,
Knack
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#287 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMr6vZq0ym1TFaQJ0a_VLLYOCYDqLIF7ks5r6W3CgaJpZM4NXhKX>
.
|
Hi Vladimir, My assumption is you're hitting the limit of your systems (as hinted in this post). Can you explain more if you use multithreading too? Knack |
Hi Knack,
We don't use multithreading yet, all calls happen in single process during
one "sync session".
That error happens not every time, but when we are creating huge amount of
campaigns (and accounts/adgroups/ads) at time.
E.g. we have 500 accounts (with ads and other stuff) to create. 200 are
being processed fine, and starting on 201, exception keeps popping up on
every API call.
We will try to increase max fopen to 4096 and test.
…On 16 May 2017 at 21:11, Thanet Knack Praneenararat < ***@***.***> wrote:
Hi Vladimir,
My assumption is you're hitting the limit of your systems (as hinted in this
post <http://stackoverflow.com/a/33447780>).
If you're hitting the rate limit
<https://developers.google.com/adwords/api/docs/guides/rate-limits> of
the AdWords API, there will be errors returned from the servers.
Can you explain more if you use multithreading too?
Knack
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#287 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMr6vR6bzzmO4E2COExV21of1CmIHLYQks5r6a55gaJpZM4NXhKX>
.
|
Hi Knack,
I think we figured out what is the issue here.
Problem is in code for Adwords session creation:
``` $filepath = APPLICATION_PATH . '/config/google/adsapi_php.ini';
$oAuth2Credential = (new OAuth2TokenBuilder())
->fromFile($filepath)
->build();
$session = (new AdWordsSessionBuilder())
->fromFile($filepath)
->withOAuth2Credential($oAuth2Credential)
->withClientCustomerId($clientCustomerId);```
It's called every time new service is created. I don't cache session object
because new "Client customer ID" is needed to be set every time.
And I don't know if I can create session object just once and then change
customer ID "on the fly". It was possible in the old library.
So looks like in our long-running script the leak comes from opening config
file without closing it.
I wonder how we can avoid that.
|
Hi Vladimir, You don't need to create a session every time you change the services to use. For example, in this line, Again, the only scenario that you have to create a new session object is when you change Best, |
Hi, Knack,
Ok, I see. But I still have to use different clientCustomerId in my script.
For example, to create an account I'm using manager account's customer ID,
then to create campaigns inside newly created account I'm using it's
customer ID.
And so on, repeat X times (500, 1000 etc), every time manager account could
be different (though totally we have 3-5 of them).
That will still generate some unclosed fopen's. But implementing session
caching by customer ID should improve situation with fopen.
But still, we noticed that config file is not released/closed correctly,
because fopen number persisted even when my script stopped running. Maybe
there's something in library code you can improve?
Vladimir.
|
Hello Vladimir,
There is no complication in the logic here, as the config file is read by the native PHP function:
To clarify, when it repeats for 500 times, you create 3-5 sessions each time? Based on what I've researched, it looks like sometimes the garbage collector is not aggressive enough and you can still run into the problem that memory isn't freed up properly.
I've tried using the above two functions in the past and it did help in some cases. Best, |
Hi Knack,
Thanks for great answer!
To clarify, when it repeats for 500 times, you create 3-5 sessions each
time?
No, that should be maximum two sessions each time - one for manager
subaccount and another one for customer account.
Well, some operation may require session created with main MCC account ID,
then three different ID's per one loop.
3-5 is the number of different manager subaccounts we have in our system.
I hope caching will help there. Also I will try to implement your other
advices about GC and variable unsetting.
We will test that this week and let you know if it solves the issue.
Vladimir.
|
Hi Vladimir, I'll try to reproduce on my side as well so I'd like to ask for more information about your scenarios. Could you please confirm the following?
Thanks in advance. |
Hi Knack,
Before we proceed with that, could I quickly ask you about another
exception I got?
It is related to Monolog:
```
Exception 'InvalidArgumentException' with message 'A stream must either be
a resource or a string.
```
```
#0
.../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/Common/AdsLoggerFactory.php(42):
Monolog\Handler\StreamHandler->__construct(false, 200)
#1
/library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/AdWords/AdWordsSessionBuilder.php(342):
Google\AdsApi\Common\AdsLoggerFactory->createLogger('AW_BATCH_JOBS_U...')
#2
/library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/AdWords/AdWordsSessionBuilder.php(297):
Google\AdsApi\AdWords\AdWordsSessionBuilder->defaultOptionals()
#3 ...: Google\AdsApi\AdWords\AdWordsSessionBuilder->build()
```
Code I use to build session:
```
$oAuth2Credential = (new OAuth2TokenBuilder())
->withClientId($clientId)
->withClientSecret($clientSecret)
->withRefreshToken($refreshToken)
->build();
$adsLoggerFactory = new AdsLoggerFactory();
$soapLogger = $adsLoggerFactory->createLogger(
'AW_SOAP',
APP_BASE_PATH . '/dynamic/logs/soap.log',
'INFO'
);
$session = (new AdWordsSessionBuilder())
->withOAuth2Credential($oAuth2Credential)
->withDeveloperToken($developerToken)
->withClientCustomerId($clientCustomerId)
->withUserAgent('Cleoo-MarketingEngine')
->withSoapLogger($soapLogger);
if ($reportSettings instanceOf ReportSettings) {
$session = $session->withReportSettings($reportSettings);
}
$session = $session->build();
```
It looks like it tries to create batchJobsUtilLogger but fails by some
reason.
Thanks in advance, Vladimir
|
Hi Vladimir, How does your It looks like Best, |
Hi Knack,
"APP_BASE_PATH" is internal constant of our application and it's always
defined at this point.
Problem is that this exception comes up not right away on script start, but
in the middle of that long-running script I described earlier.
And exception trace shows that it's not generated from my code, as I'm
creating "soapLogger":
``` $adsLoggerFactory->createLogger(
'AW_SOAP',
```
And in exception "batchJobsUtilLogger" is being created at
AdWordsSessionBuilder.php:342
```
Google\AdsApi\Common\AdsLoggerFactory->createLogger('AW_BATCH_JOBS_U...')
```
Vladimir.
|
Regarding your previous questions, I'm still collection system infromation,
but will answer on some of them.
Our PHP version is 7.0.1, OS is Linux (Ubuntu).
In your scenario, could you please share a bit more about the following?
You says your repeat the process for X times, e.g., X = 500, 1000. What's
the maximum of this X?
We have an MCC account with tsousands of customer accounts inside. As I
remember it's about 4'000 in biggest one.
Provided that totally you have not so many sessions objects in the
system, in each iteration, the factor that you're changing is only
services, e.g., from using AdGroupService to CampaignService? Are there any
factors you're varying here?
Well, as I have to create session for each customer ID, there will be
number of sessions equal to count of customer accounts, so 4'000+.
What do you do in each iteration? Services get()? mutate()? or both?
In that script I use CampaignService, AdGroupService
and AdGroupCriterionService, only "get()" methods.
Point is simple:
1) Getting list of all cutomer accounts of given MCC
2) For every account, getting list of campaigns
3) For every campaign getting ad groups
4) For every ad group gettings ads and keywords
Everything is saved into database.
You mentioned that it looks like some files are not closed after your
script ends. How long does it take to close? Or are they never closed and
you need to kill them manually each time?
Looks like I misunderstood and files are closed. And I've changed part
where session is created to not use configuration file but strings from
database.
Vladimir.
|
Hi Vladimir, Thanks for more insights! One question I have for your explanation:
But they all don't persist at the same time, right? Totally, there shouldn't be 4,000 session objects existing at the same time? And about your usage pattern, it'd be much better to use structure reports. Cheers, |
Hi Vladimir, As for your earlier question, looking more closely, I see that the issue lies in the initialization of batch job logger. But somehow, Best, |
Hi Knack,
All code I used to create session I've posted above. I don't configure batch
job logger and I don't use configuration file.
I guess it will be fixed once I'll create and use one.
Vladimir.
|
Hi Knack,
After all fixes it failed in the middle again with old error:
Exception 'SoapFault' with message 'SOAP-ERROR: Parsing WSDL: Couldn't load
from '
https://adwords.google.com/api/adwords/mcm/v201702/ManagedCustomerService?wsdl'
: failed to load external entity "
https://adwords.google.com/api/adwords/mcm/v201702/ManagedCustomerService?wsdl
"
Stack trace:
#0 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/Common/AdsSoapClient.php(65):
SoapClient->SoapClient('https://adwords...', Array)
#1 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/AdWords/v201702/mcm/ManagedCustomerService.php(91):
Google\AdsApi\Common\AdsSoapClient->__construct('https://adwords...',
Array)
#2 [internal function]:
Google\AdsApi\AdWords\v201702\mcm\ManagedCustomerService->__construct(Array)
#3 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/Common/Util/Reflection.php(39):
ReflectionClass->newInstanceArgs(Array)
#4 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/Common/AdsSoapClientFactory.php(87):
Google\AdsApi\Common\Util\Reflection->createInstance('Google\\AdsApi\\A...',
Array)
#5 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/AdWords/AdWordsServices.php(55):
Google\AdsApi\Common\AdsSoapClientFactory->generateSoapClient(Object(Google\AdsApi\AdWords\AdWordsSession),
Object(Google\AdsApi\AdWords\AdWordsHeaderHandler),
Object(Google\AdsApi\AdWords\AdWordsServiceDescriptor))
#6 ...: Google\AdsApi\AdWords\AdWordsServices->get(Object(Google\AdsApi\AdWords\AdWordsSession),
'Google\\AdsApi\\A...')
Does it tell you anything? We had it already (I've created an issue here),
but I found no solution, we just changed host where we run it. Look like
all that is related somehow (CURL errors before and this error).
Vladimir.
…On 25 May 2017 at 15:14, Vladimir Tatarsky ***@***.***> wrote:
Hi Knack,
All code I used to create session I've posted above. I don't configure batch
job logger and I don't use configuration file.
I guess it will be fixed once I'll create and use one.
Vladimir.
|
Hello Vladimir, Does this happen all the time (you can't get through at all) or intermittently after many API requests? Thanks. |
Hi Knack.
Second option, after many API requests.
Vladimir.
…On 25 May 2017 at 23:57, Thanet Knack Praneenararat < ***@***.***> wrote:
Hello Vladimir,
Does this happen all the time (you can't get through at all) or
intermittently after many API requests?
Thanks.
Knack
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#287 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMr6veD6V2sRqAbUxTY2kMDE6rtTklAMks5r9bMVgaJpZM4NXhKX>
.
|
Hi Knack,
We figured out that prior to "Parsing WSDL" exception CURL error occured
(same as before):
`Exception 'GuzzleHttp\Exception\ConnectException' with message 'cURL error
35: error:02001018:system library:fopen:Too many open files (see
http://curl.haxx.se/libcurl/c/libcurl-errors.html)`
So I think it's not really problem of this library, but Guzzle problem,
which don't have real solution yet.
Such problem occurs with other SDK's too, e.g.:
https://forums.aws.amazon.com/thread.jspa?threadID=181885
And I guess there is no possibility to switch to using CURL (as in old
library) for non-SOAP HTTP API calls.
I'd successfully ran same code with old library version without hitting any
limits.
So now only thing I can do is to refactor my code to implement some kind of
multithreading.
Vladimir.
|
Hi Knack, I've created an issue in Guzzle and library author answered: #1846
(comment)
<guzzle/guzzle#1846 (comment)> Can
you check if $response->getBody()->close() is used in your library? I mean
if stream is closed properly for services. Vladimir.
|
Hi Vladimir, Thanks for helping investigate this issue. Where did you get the below error?
The library's SOAP services don't make use of the Guzzle, so I wonder how the above error is generated. Best, |
Hi Knack,
Full exception trace is in the first post of the issue.
Here is the part between our app and Guzzle:
#12 .../library/vendor/guzzlehttp/guzzle/src/Client.php(104):
GuzzleHttp\Client->sendAsync(Object(GuzzleHttp\Psr7\Request), Array)
#13 .../library/vendor/google/auth/src/HttpHandler/Guzzle6HttpHandler.php(34):
GuzzleHttp\Client->send(Object(GuzzleHttp\Psr7\Request), Array)
#14 .../library/vendor/google/auth/src/OAuth2.php(492):
Google\Auth\HttpHandler\Guzzle6HttpHandler->__invoke(Object(GuzzleHttp\Psr7\Request))
#15 .../library/vendor/google/auth/src/Credentials/UserRefreshCredentials.php(92):
Google\Auth\OAuth2->fetchAuthToken(Object(Google\Auth\HttpHandler\Guzzle6HttpHandler))
#16 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/Common/Util/OAuth2TokenRefresher.php(60):
Google\Auth\Credentials\UserRefreshCredentials->fetchAuthToken()
#17 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/AdWords/AdWordsHeaderHandler.php(72):
Google\AdsApi\Common\Util\OAuth2TokenRefresher->getOrFetchAccessToken(Object(Google\Auth\Credentials\UserRefreshCredentials))
#18 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/Common/AdsSoapClient.php(86):
Google\AdsApi\AdWords\AdWordsHeaderHandler->generateHttpHeaders(Object(Google\AdsApi\AdWords\AdWordsSession))
#19 .../library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/AdWords/v201702/mcm/ManagedCustomerService.php(103):
Google\AdsApi\Common\AdsSoapClient->__soapCall('get', Array)
#20 .../.../OurAppService.php(237):
Google\AdsApi\AdWords\v201702\mcm\ManagedCustomerService->get(Object(Google\AdsApi\AdWords\v201702\cm\Selector))
Looks like it happens when it's trying to call"get" method of some service
(not neccecary exactly ManagedCustomerService - it could be any).
Exactly when it tries to refresh OAuth credentials:
google/auth/src/Credentials/UserRefreshCredentials.php(92):
Google\Auth\OAuth2->fetchAuthToken
Vladimir.
|
Hi Vladimir, I see. That happens during the token refreshing process, which is a part of Google Auth library, not during the SOAP calls. Let me think how we can resolve this. By the way, is it possible to have your curl version as well? Thanks. |
Hi Knack,
Just sending some details you asked before.
Curl version: curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0
OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
PHP version: 7.0.17-2
default_socket_timeout was 60, changed it to 300 and it didn't helped.
Linux vesion: Ubuntu 14.04.3 LTS
Vladimir.
… |
Hi Knack,
I see some fix was merged recently, how soon it will be included in
release? Or should we just try to pull latest master branch?
Vladimir.
|
Hi Vladimir, I've just made a release for v27.1.0, which also includes the latest version of the API (v201705). I'll close this issue for now. Feel free to report if you find other different issues. Cheers, |
Hi guys, I've just got this error:
#0 /library/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(150): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array) #1 /library/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(103): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory)) #2 /library/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(43): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory)) #3 /library/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #4 /library/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(51): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #5 /library/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(72): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #6 /library/vendor/guzzlehttp/guzzle/src/Middleware.php(30): GuzzleHttp\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #7 /library/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(68): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #8 /library/vendor/guzzlehttp/guzzle/src/Middleware.php(59): GuzzleHttp\RedirectMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #9 /library/vendor/guzzlehttp/guzzle/src/HandlerStack.php(67): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #10 /library/vendor/guzzlehttp/guzzle/src/Client.php(275): GuzzleHttp\HandlerStack->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #11 /library/vendor/guzzlehttp/guzzle/src/Client.php(96): GuzzleHttp\Client->transfer(Object(GuzzleHttp\Psr7\Request), Array) #12 /library/vendor/guzzlehttp/guzzle/src/Client.php(104): GuzzleHttp\Client->sendAsync(Object(GuzzleHttp\Psr7\Request), Array) #13 /library/vendor/google/auth/src/HttpHandler/Guzzle6HttpHandler.php(34): GuzzleHttp\Client->send(Object(GuzzleHttp\Psr7\Request), Array) #14 /library/vendor/google/auth/src/OAuth2.php(492): Google\Auth\HttpHandler\Guzzle6HttpHandler->__invoke(Object(GuzzleHttp\Psr7\Request)) #15 /library/vendor/google/auth/src/Credentials/UserRefreshCredentials.php(92): Google\Auth\OAuth2->fetchAuthToken(Object(Google\Auth\HttpHandler\Guzzle6HttpHandler)) #16 /library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/Common/Util/OAuth2TokenRefresher.php(60): Google\Auth\Credentials\UserRefreshCredentials->fetchAuthToken() #17 /library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/AdWords/AdWordsHeaderHandler.php(72): Google\AdsApi\Common\Util\OAuth2TokenRefresher->getOrFetchAccessToken(Object(Google\Auth\Credentials\UserRefreshCredentials)) #18 /library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/Common/AdsSoapClient.php(86): Google\AdsApi\AdWords\AdWordsHeaderHandler->generateHttpHeaders(Object(Google\AdsApi\AdWords\AdWordsSession)) #19 /library/vendor/googleads/googleads-php-lib/src/Google/AdsApi/AdWords/v201702/mcm/ManagedCustomerService.php(103): Google\AdsApi\Common\AdsSoapClient->__soapCall('get', Array) #20 /home/ubuntu/cleoo/application/Modules/marketingengine/src/Service/NewAdwords/Helpers/AccountHelper.php(62): Google\AdsApi\AdWords\v201702\mcm\ManagedCustomerService->get(Object(Google\AdsApi\AdWords\v201702\cm\Selector)) ...
Any ideas how it could be fixed?
The text was updated successfully, but these errors were encountered: