This has me fed up: I started to add an observation about 20:30 GMT today using a new iPad. Hung up, spinning thing and no actions allowed, but only hung up when obs was almost complete at the stage of adding the sci3ntific name, so efforts wasted. Eventually black screen with 500 message. Tried forum, no entry allowed, still spinning. Later on, still no entry though tried to reload. Had a quick look a few mins ago, I am restored though was still logged in but new obs construction lost.
This hanging up has happened several times in between this and my last report, often using phone (iPhone 12) and this new iPad. Disheartening, please fix this, other sites are normal.
Looks as if things are working for me - no problems, since a tiny hesitation early today (9th Mar) - so things are looking up Thanks - or am I speaking too soon?
I had the same experience as Mags yesterday but then it all seemed to be OK again in the evening. I agree though that it is frustrating if you have waited ages for several photos to upload and then it fails over something as simple as looking for a name in the dictionary. It probably explains why iSpot is down to a hard core of perhaps 35 (?) regular users.
Hello. Looks like your robots.txt file has gone missing. I know there was one before. If you try to load it youāll also see that there is no branded 404 page.
This isnāt going to help from a performance point of view (depending on how sophisticated the crawlers are) as there are some pages/actions, commonly linked by search filters, that perform badly.
If those pages are putting load on the DB servers leading to a queue of requests, that will pretty much destroy the performance of everything else in the processā¦ leading to errors on any page that requires DB activity (sessions, looking up user data etc.) and if the Forum uses the same server/tables for anything (directly or indirectly) youāll hit the same problem there too when this happens.
I donāt know the ins and outs of your setup, but this would explain why thereās no pattern with respect to browsers or operating systems, i.e. itās not the client here. Iām sure once the site starts tipping over, even requests from text based browsers will failā¦ until everything/everyone backs off and it returns to normal.
Hiding pages from crawlers (who might not always obey the rules anyway) may create other issues (e.g. SEO), but I guess the priority here is to get the site running reliably for your current users. That then buys some time (depending on whatās left) to deal with performance issues relating to the underlying queries.
Yes Ken I was getting the same impression, not many regulars. The ZA community regulars all migrated to iNat with the problems we had in 2018 or thereabout - Mike will remember.
Meanwhile, if/when your upload stalls, just complete the bits you are able, and check another tab to see if iSpot is stalling, leave your work on hold and if necessary wait overnight - Iām using Windows and am able to Hibernate instead of turning off - the next day Iām able to pick up where I left off. Derek also taught me to use a text file as a ācarbon copyā.
Iād be very sad to see iSpot fail, I have put a lot of work into my observations, and still think itās a fantastic site.
Maybe we need to remember Keep Calm and Carry On
And again. But I did manage to add an obs, the 4 th time I tried this one. Things looked good, not long logged in and had visited a project and canāt remember what else before making the observation. Then wanted to edit the observation just made, I changed text a little and the operation would not complete when confirmation invoked. Spinning for a few minutes followed by red banner saying observation could not be updated.
New iPad, about 12 noon GMT today.
Editing to add: checked back about 10 mins later, still excluded, spinning thing, limited display of page contents. Then the black 500 error screen. Re Using my devices iPhone or iPad, I donāt see any workarounds mentioned by those who understand this stuff. I wonder how those with access only via eg a phone are managing and could I learn from them.
How did you spot the lack of robots.txt? Programmers checked and it was not showing for a time but then back again - all without the pages having been touched so this is rather odd.
I remembered there was previously one and I knew when I examined it that there were attempts to prevent certain pages being crawled and so after checking the average performance of various types of pages I looked to see if they were being excludedā¦ and there was no robots.txt
If itās sometimes there and sometimes not then it sounds like you may have issues with keeping the contents of multiple servers matchedā¦ or some bizarre cache management problem.
Just tried to access robots.txt again and Iām still getting 404.
If a crawler looks for one firstā¦ before, say, offloading the job of fetching pages across multiple threads then thatās potentially bad news.
Just this minute, 8:45 GMT. New iPad, Safari.
Had just added a comment to the Mustelid obs by Miked, then went back to look at some other observations, but spinning thing appeared and would not stop.
Edited to add: still spinning when I went back to the site. No workarounds for such a device, so it just seems to take time to resolve.
Further edit, checked back again, now the black 500 error screen. Annoying.
I find that I need to try iSpot only after closing/reopening Firefox (or deleting all browsing data). Even a relatively small cache (in the last instance, 12.7 Mb) can cause problems.
Surely, any new users - and perhaps some old hands - will find this ongoing problem a deterrent.
Hello. Hereās a summary of what I think is going onā¦
There are at least two servers that provide the main iSpot web site. Likely load balanced by Cloudflare and probably with sticky sessions.
Iāll assume that iSpot can cope with ordinary levels of traffic as generated by humans on their browsers.
At least one of the servers is missing a robots.txt file. I know this because I can run two machines side-by-side and find that while one is handed a robots.txt file, the other can find itself receiving the ādefault backend - 404ā error page.
At some point crawlers (1 or more) which would normally be told NOT to crawl the site will stumble upon the one without a robots.txt file and proceed to wade through ALL of its pages. When this happens will be entirely random and none of the rules in the robots.txt file will applyā¦ because it wasnāt there at the time the request was made.
The resulting traffic brings that server to its knees.
Anyone who Cloudflare has decided should be directed to that same server (due to sticky sessions) eventually experiences the 500 error as the server is overwhelmed.
Those who delete all their cookies/data will then be treated by Cloudflare as a new visitor. If deleting cookies/data ALWAYS works then its likely because Cloudflare, by this point, has marked that server as bad and so directs ALL new traffic to the other servers.
If, however, the database server is also struggling then I suspect all clients will be affected until the crawler (hopefully) determines that something is wrong and backs off.
Whether one or ALL of the servers becomes non-responsive I suspect depends on the nature of the pages being accessed by the crawler.
So I would say that the first step would be to get the robots.txt file hosted on ALL the servers (and it might be an idea to review the deployment pipeline) and then set about improving the performance of the slowest pages. This is still necessary because crawlers donāt have to obey the rules in the robots.txt file anyway.
ā¦ and I should also add that itās entirely possible that any of the available web servers are potential targets once the crawler thinks it can go ahead. Itāll all be down to implementation.
