[ask] Cara setting robots.txt?

Discussion in 'General Internet' started by ziuma, Oct 24, 2017.

  1. ziuma

    ziuma Well-Known Member

    Joined:
    May 23, 2014
    Messages:
    1,442
    Likes Received:
    212
    Trophy Points:
    63
    Google+:
    Selamat Malam semua,

    Saya mau tanya sedikit, mudah-mudahan dapat jawaban yang banyak :D
    bagaimana ya cara setting robots.txt

    supaya robots.txt tersembunyi dari pengunjung tetapi bot dari search engine masih bisa membacanya

    maksud saya: robots.txt hanya untuk bot search engine aja

    Terima Kasih untuk bantuannya
     
  2. Dadan rusdan

    Dadan rusdan Super Level

    Joined:
    May 29, 2014
    Messages:
    2,076
    Likes Received:
    252
    Trophy Points:
    83
    Saya malah gak pernah seting karena gak tau caranya *boohoo* kalau salah seting bisa berpengaruh katanya ,begitupun kalo benar.. haha
     
  3. rosyad

    rosyad Active Member

    Joined:
    Dec 30, 2016
    Messages:
    1,108
    Likes Received:
    84
    Trophy Points:
    48
  4. ys. herbi

    ys. herbi Well-Known Member

    Joined:
    Mar 6, 2016
    Messages:
    1,240
    Likes Received:
    189
    Trophy Points:
    63
    Google+:
    Setting robot.txt.. saya pernah sih, tp ndak tau bagusnya gimana...
    Soalnya dulu pernah salah nyebut, ndak diindeks SE tertentu (lupa namanya)...

    Kok bisa tau mas?
    Lha abis nglepas robot.txt itu, ada SE "asing" yang banyak ngerujuk ke blog saya..
     
  5. KangAndre

    KangAndre Member

    Joined:
    Jan 25, 2014
    Messages:
    9,886
    Likes Received:
    2,592
    Trophy Points:
    313
    jangan buat atau hapus robot.txt, robot akan tetap meng-crawl

    Robot.txt diperlukan sebagai aturan bagi robot biar gak main selonong saja, namun ada juga robot yang "nakal" meskipun ada robot.txt tetap main serobot.
     
  6. ziuma

    ziuma Well-Known Member

    Joined:
    May 23, 2014
    Messages:
    1,442
    Likes Received:
    212
    Trophy Points:
    63
    Google+:
    ngak bisa di hide ya @KangAndre ?
    artinya robots.txt nya ada tetapi pengunjung tidak bisa lihat

    saya pasang kode seperti ini di .hattacces

    Code:
    <Files robots.txt>
    order allow,deny
    deny from all
    </Files>
    
    ternyata di google webmaster dianggap tidak ada robots.txt
     
  7. KangAndre

    KangAndre Member

    Joined:
    Jan 25, 2014
    Messages:
    9,886
    Likes Received:
    2,592
    Trophy Points:
    313
    Tidak disarankan membuat batasan robots di .htaccess.
    Alasannya:
    .htaccess digunakan untuk akses internal , sedangkan robots.txt untuk akses eksternal .

    "Internal" karena .htaccess bertugas untuk memberitahu server (Apache) bagaimana menangani nama halaman dan file, URL dan cara pengguna untuk mengakses sumber daya ini( untuk menangani fitur-fitur situs sendiri). Salah pengisian situs bisa error.

    Sedangkan Robots.txt, sebaliknya mengatur akses "eksternal", karena ini memberi tahu mesin pencari dan alat web lainnya yang harus dibaca dan yang boleh diindeks. Jika salah, hanya berakibat tidak terindeks saja, tapi situs tetap berjalan.

    Sudah ada ketentuan bahwa robots.tst dapat dibaca manusia (pengguna) dan robot.

    ziuma.com/robots.txt terbaca, coba hapus yang di .htaccess dan uji kembali via webmaster tools.
     
    ziuma likes this.
  8. davidrahman

    davidrahman Member

    Joined:
    Jun 4, 2014
    Messages:
    881
    Likes Received:
    31
    Trophy Points:
    28
    sepertinya ada hal yang penting nih di robots.txt punya ts :D
     
  9. KangAndre

    KangAndre Member

    Joined:
    Jan 25, 2014
    Messages:
    9,886
    Likes Received:
    2,592
    Trophy Points:
    313
    Semua situs bisa dilihat robots.txt-nya
    namadomain.com/robots.txt dan nggak ada rahasia penting :D
     
  10. ncang

    ncang Super Level

    Joined:
    Feb 7, 2013
    Messages:
    4,622
    Likes Received:
    747
    Trophy Points:
    113
    Google+:
    ada yang keren juga isinya,
    Code:
    https://www.nike.com/robots.txt
    https://www.google.com/killer-robots.txt
    
    ini barusan screenshot, gak pake :D
    [​IMG]
     
    ziuma likes this.
  11. ziuma

    ziuma Well-Known Member

    Joined:
    May 23, 2014
    Messages:
    1,442
    Likes Received:
    212
    Trophy Points:
    63
    Google+:
    ada logo nike dibawahnya
    keren keren :D
     
  12. KangAndre

    KangAndre Member

    Joined:
    Jan 25, 2014
    Messages:
    9,886
    Likes Received:
    2,592
    Trophy Points:
    313
    Tapi jarang orang lihat robots.txt (paling iseng saja) :D
     
  13. Sugeng Ginanjar

    Sugeng Ginanjar Member

    Joined:
    Nov 18, 2016
    Messages:
    215
    Likes Received:
    10
    Trophy Points:
    18
  14. Andre

    Andre Member

    Joined:
    Apr 12, 2013
    Messages:
    483
    Likes Received:
    21
    Trophy Points:
    18
  15. miftah666

    miftah666 New Member

    Joined:
    Sep 6, 2016
    Messages:
    23
    Likes Received:
    0
    Trophy Points:
    3
    Mendingan jangan di hapus Gan.....menurut saya
     
  16. Fathoni

    Fathoni Member

    Joined:
    Mar 9, 2016
    Messages:
    160
    Likes Received:
    11
    Trophy Points:
    18
    ikutan nyimak gan biar semakin pintar dan bertambah ilmu lagi
    :D
     
  17. Nina Ayunine

    Nina Ayunine New Member

    Joined:
    Nov 2, 2017
    Messages:
    27
    Likes Received:
    4
    Trophy Points:
    3
    tanya bawah ane mungkin paham
     
  18. Fathoni

    Fathoni Member

    Joined:
    Mar 9, 2016
    Messages:
    160
    Likes Received:
    11
    Trophy Points:
    18
  19. Nastriawan

    Nastriawan Member

    Joined:
    Sep 7, 2016
    Messages:
    61
    Likes Received:
    6
    Trophy Points:
    8
    ijin nyimak gan, sy juga msh belum ngerti
     
Loading...

Share This Page