Error 500 + non-responsive website report log

This has me fed up: I started to add an observation about 20:30 GMT today using a new iPad. Hung up, spinning thing and no actions allowed, but only hung up when obs was almost complete at the stage of adding the sci3ntific name, so efforts wasted. Eventually black screen with 500 message. Tried forum, no entry allowed, still spinning. Later on, still no entry though tried to reload. Had a quick look a few mins ago, I am restored though was still logged in but new obs construction lost.

This hanging up has happened several times in between this and my last report, often using phone (iPhone 12) and this new iPad. Disheartening, please fix this, other sites are normal.

Looks as if things are working for me - no problems, since a tiny hesitation early today (9th Mar) - so things are looking up Thanks - or am I speaking too soon?

I had the same experience as Mags yesterday but then it all seemed to be OK again in the evening. I agree though that it is frustrating if you have waited ages for several photos to upload and then it fails over something as simple as looking for a name in the dictionary. It probably explains why iSpot is down to a hard core of perhaps 35 (?) regular users.

1 Like

Hello. Looks like your robots.txt file has gone missing. I know there was one before. If you try to load it youā€™ll also see that there is no branded 404 page.

This isnā€™t going to help from a performance point of view (depending on how sophisticated the crawlers are) as there are some pages/actions, commonly linked by search filters, that perform badly.

If those pages are putting load on the DB servers leading to a queue of requests, that will pretty much destroy the performance of everything else in the processā€¦ leading to errors on any page that requires DB activity (sessions, looking up user data etc.) and if the Forum uses the same server/tables for anything (directly or indirectly) youā€™ll hit the same problem there too when this happens.

I donā€™t know the ins and outs of your setup, but this would explain why thereā€™s no pattern with respect to browsers or operating systems, i.e. itā€™s not the client here. Iā€™m sure once the site starts tipping over, even requests from text based browsers will failā€¦ until everything/everyone backs off and it returns to normal.

Hiding pages from crawlers (who might not always obey the rules anyway) may create other issues (e.g. SEO), but I guess the priority here is to get the site running reliably for your current users. That then buys some time (depending on whatā€™s left) to deal with performance issues relating to the underlying queries.

1 Like

Yes Ken I was getting the same impression, not many regulars. The ZA community regulars all migrated to iNat with the problems we had in 2018 or thereabout - Mike will remember.
Meanwhile, if/when your upload stalls, just complete the bits you are able, and check another tab to see if iSpot is stalling, leave your work on hold and if necessary wait overnight - Iā€™m using Windows and am able to Hibernate instead of turning off - the next day Iā€™m able to pick up where I left off. Derek also taught me to use a text file as a ā€˜carbon copyā€™.
Iā€™d be very sad to see iSpot fail, I have put a lot of work into my observations, and still think itā€™s a fantastic site.
Maybe we need to remember Keep Calm and Carry On

1 Like

And again. But I did manage to add an obs, the 4 th time I tried this one. Things looked good, not long logged in and had visited a project and canā€™t remember what else before making the observation. Then wanted to edit the observation just made, I changed text a little and the operation would not complete when confirmation invoked. Spinning for a few minutes followed by red banner saying observation could not be updated.
New iPad, about 12 noon GMT today.

Editing to add: checked back about 10 mins later, still excluded, spinning thing, limited display of page contents. Then the black 500 error screen. Re Using my devices iPhone or iPad, I donā€™t see any workarounds mentioned by those who understand this stuff. I wonder how those with access only via eg a phone are managing and could I learn from them.

How did you spot the lack of robots.txt? Programmers checked and it was not showing for a time but then back again - all without the pages having been touched so this is rather odd.

I remembered there was previously one and I knew when I examined it that there were attempts to prevent certain pages being crawled and so after checking the average performance of various types of pages I looked to see if they were being excludedā€¦ and there was no robots.txt

If itā€™s sometimes there and sometimes not then it sounds like you may have issues with keeping the contents of multiple servers matchedā€¦ or some bizarre cache management problem.

Just tried to access robots.txt again and Iā€™m still getting 404.

If a crawler looks for one firstā€¦ before, say, offloading the job of fetching pages across multiple threads then thatā€™s potentially bad news.

I tried it between Mikeā€™s and your posts, and found it present. (All crawlers other than Google nominally banned.)

2025-03-12 1445UTC Trying to load The striped one | Observation | UK and Ireland | iSpot Nature. Cleared cookies and all worked properly. Windows10 Chrome.

1649 On opening ispotnature.org/communities/uk-and-ireland/view/observation/869718/whorled-mint from changes tracker

2025-03-15, 0755UTC. Trying to add observation. Windows 10; Chrome. Cleared cookies then successfully added observation.

Just this minute, 8:45 GMT. New iPad, Safari.
Had just added a comment to the Mustelid obs by Miked, then went back to look at some other observations, but spinning thing appeared and would not stop.
Edited to add: still spinning when I went back to the site. No workarounds for such a device, so it just seems to take time to resolve.
Further edit, checked back again, now the black 500 error screen. Annoying.

Have you tried deleting iSpot cookies? This seems to resolve the problem for me every time.

That can work sometimes on desktop for me, no idea how to do that on phone or iPad though

Settings ā†’ Apps ā†’ app_name

ā€¦ is where youā€™ll probably find something useful.

I find that I need to try iSpot only after closing/reopening Firefox (or deleting all browsing data). Even a relatively small cache (in the last instance, 12.7 Mb) can cause problems.
Surely, any new users - and perhaps some old hands - will find this ongoing problem a deterrent.

2 Likes

Yes!
And as this post requires more emphasis, imagine Yes x 20 characters!

Hello. Hereā€™s a summary of what I think is going onā€¦

There are at least two servers that provide the main iSpot web site. Likely load balanced by Cloudflare and probably with sticky sessions.

Iā€™ll assume that iSpot can cope with ordinary levels of traffic as generated by humans on their browsers.

At least one of the servers is missing a robots.txt file. I know this because I can run two machines side-by-side and find that while one is handed a robots.txt file, the other can find itself receiving the ā€œdefault backend - 404ā€ error page.

At some point crawlers (1 or more) which would normally be told NOT to crawl the site will stumble upon the one without a robots.txt file and proceed to wade through ALL of its pages. When this happens will be entirely random and none of the rules in the robots.txt file will applyā€¦ because it wasnā€™t there at the time the request was made.

The resulting traffic brings that server to its knees.

Anyone who Cloudflare has decided should be directed to that same server (due to sticky sessions) eventually experiences the 500 error as the server is overwhelmed.

Those who delete all their cookies/data will then be treated by Cloudflare as a new visitor. If deleting cookies/data ALWAYS works then its likely because Cloudflare, by this point, has marked that server as bad and so directs ALL new traffic to the other servers.

If, however, the database server is also struggling then I suspect all clients will be affected until the crawler (hopefully) determines that something is wrong and backs off.

Whether one or ALL of the servers becomes non-responsive I suspect depends on the nature of the pages being accessed by the crawler.

So I would say that the first step would be to get the robots.txt file hosted on ALL the servers (and it might be an idea to review the deployment pipeline) and then set about improving the performance of the slowest pages. This is still necessary because crawlers donā€™t have to obey the rules in the robots.txt file anyway.

I hope thatā€™s useful.

ā€¦ and I should also add that itā€™s entirely possible that any of the available web servers are potential targets once the crawler thinks it can go ahead. Itā€™ll all be down to implementation.