the instance list is gone when i delete yesterday data

[taotoxht]

Dear Team,

I am use skywalking on production, Because the elasticsearch disk is full, so i write a linux crontab job delete yesterday data every day at 6：00 pm。then the instance list is gone from the Instance tab of General-Service dashboard。 like below screenshot

in the overview tab the data is normal：

After waiting all day, it didn't recover ， must i restart the client agent, it could be ok? but i am on production env, Is there any other way to solve it?

Thanks and Regards,

[wu-sheng]

Are you making the crontab keep running? Ideally, the instance would be regenerated if it has traffic/metrics.

More importantly, removing yesterday's data is a very bad practice. Don't do that.

You could reboot the whole OAP server cluster, and I believe the data would be bad. Of course, make sure you would not create a crontab like this.

[taotoxht]

since I reboot the oap , i recoverd.

[taotoxht]

I'm curious why the heart beat exists,why it can't regenerate instance_traffic. I found from this page below：
#3453

I stop the crontab and waiting all day，I thought it would recover automatically，but it didn't 。 thank you~

[wu-sheng]

I don't know either. You may remove the index template, rather than the indices. But I am not guessing.

[taotoxht]

My crontab job execute below shell：
curl --location --request DELETE 'http://10.153.99.7:9200/*-20221011*' --header 'Accept: application/json'  --header 'Authorization: Basic ZWxhc3RpYzpnaHdjdUVzVj13U3lrLV9WWF9GNw==' --header 'Cache-Control: no-cache' --header 'Content-Type: application/json' 
does this can delete oap es temlapte？
but i can't found oap elastic temlpate by es api：http://10.153.99.7:19200/_template， only this

thank you~

[wu-sheng]

I can't tell what happens on your site, but SkyWalking definitely has index templates.

[taotoxht]

i kwnow why i can't find the index template. I change 'http://10.153.99.7:19200/_template' to 'http://10.153.99.7:9200/_index_template/xht01*' ,then get the result as below:

but i still don't know why the instance not regenerated ，i am sure there is a lot of traffic as shown in the screenshot above

[sruby]

I had the same problem because our operator cleared ES data through crontab jobs.
Fix options:
Stop crontab jobs, stop SkyWalking, clear ES data and start SkyWalking.